Adaptive Control based on Adjustable Evaluation Function

Every control design involves manipulating a controller so that a controlled plant behaves as desired. Various methodologies for manipulating the controller for certain well-defined classes of control problems have been well developed and proven. Despite great efforts, similar techniques are not available for more general classes of control problems involving nonlinear systems. Incorporating adaptive methods for manipulating the controller has been a widely used alternative. In many adaptive control system designs, the control objective is defined as to minimize the plant output error (i.e., the error between a goal and an actual plant state), assuming that smaller plant output errors imply better instantaneous control performances or immediate rewards. However, in many control problems, executing the actions that are predicted to result in smaller plant output errors only can mislead to non-goal states. This is because smaller plant output errors do not always imply better instantaneous control performances, i.e., the plant output error is not always an evaluative information. In Reinforcement Learning (RL), the evaluative information about action performances is assumed not readily available. In contrast with a customary in adaptive control designs where the actions are rewarded based on the control performance measure defined a priori by an engineer, RL views that the actions must be rewarded by the environment, not by the engineer. RL rewards the actions based on a value function, i.e, a function that represents ”goodness” of a state or a state-action. The value function is learned only from the long-term consequences of executing actions (i.e., being given a reward or a penalty) in every iteration, and then used to decide control actions. A precise value function simply recommends to take an action that can lead to